Evaluating conversational recommender systems

Abstract Conversational recommender systems aim to interactively support online users in their information search and decision-making processes an intuitive way. With the latest advances voice-controlled devices, natural language processing, AI general, such received increased attention recent years. Technically, conversational recommenders are usually complex multi-component applications often consist of multiple machine learning models a user interface. Evaluating system holistic way can therefore be challenging, as it requires (i) assessment quality different components, (ii) perception whole by users. Thus, mixed methods approach is required, which may combine objective (computational) subjective (perception-oriented) evaluation techniques. In this paper, we review common approaches for systems, identify possible limitations, outline future directions towards more practices.